Hybrid Machine Learning-Based Approach for Anomaly Detection using Apache Spark

Over the past few decades, volume of data has increased significantly in both scientific institutions and universities, with a large number students enrolled high related data. Furthermore, network traffic post-pandemic use online learning. Therefore, processing is complex challenging task that increases possibility intrusions anomalies. Traditional security systems cannot deal such high-speed big traffic. Real-time anomaly detection should be able to process as quickly possible detect abnormal malicious This paper proposes hybrid approach consisting supervised unsupervised learning for based on engine Apache Spark. Initially, k-means algorithm was implemented Sparks MLlib clustering traffic, then each cluster, K-nearest neighbors (KNN) classification detection. The proposed model trained validated against real dataset from Ibn Zohr University. results indicate outperformed other well-known algorithms detecting anomalies aforementioned dataset. experimental show can reach up 99.94 % accuracy using k-fold cross-validation method complete all 48 features.